Hi @antgonza ,

I just realized that plugin tests behave slightly different than I expected:

1. Tests often do not read the plugin configuration file, as the "commands" are directly called, instead of creating a now plugin job. Thus, a different "plugin_coupling" protocol - as stored in the config file - is not properly used for the test. To fix this, I suggest that the PluginTestCase class of the qiita_client also checks the environment variable QIITA_PLUGINCOUPLING. Thus, by setting this env, tests will enable different protocols even though they don't adhere to the value in the generated plugin configuration file. (This PR)
2. Resetting the DB of qiita is done by triggering the /apitest/reset/ endpoint (and also others). However, this only works properly if qiita master is addressed. Tests therefore do not speak against nginx (port 8383) but directly with the master instance (port 21174). However, when sending files, I relied on delivery through nginx. If requests go directly to master, the file content will be empty - and tests fail. With PR Tornado fetch file from central handler qiita#3483 the according endpoint checks if a request was made through nginx (and then uses it's fast mechanism to deliver the file) or directly to the tornado instance (which now can also send the file in a much slower fashion). Should be fast enough for testing.

As we use qiita_client in qp-target-gene, which operates on old python2, we need to make the makedirs function compatible. The argument exist_ok did not exist in py2.

I've also added the ability to deposit not only a single file, but - if user provides a directory - recurse through this directory and iteratively deposit all found files. This is needed e.g. for qp-target-gene artifacts: https://github.com/qiita-spots/qp-target-gene/blob/858436ee6d4a1fccd35faee0b156a12953b5774a/qp_target_gene/pick_otus.py#L124

Similarly, clients can now fetch directories. As I fear that misbehaving clients might trigger fetching the whole QIITA_BASE_DIR, I am limiting this to those directories that are "managed" by Qiita's DB as filetype directory.

I realized that https://github.com/qiita-spots/qtp-sequencing/blob/4a325e7a75ca989eea9d9029acd02be2384a601e/qtp_sequencing/validate.py#L81 is directly deleting files in QIITA_BASE_DIR. Therefore, I added an according function to qiita_client and qiita, but only delete the file when using "filesystem" protocol OR qiita is run in test mode as I am too skeptical about providing an API endpoint to delete arbitrary files.